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Abstract 



We conducted systematic observations of the 
implementation of a videodisc-based, direct instruction 
program in fractions.* The nine upper-elementary 
classrooms involved were spread across three school 
districts. " Implementation/utilization of the program 
varied across classrooms, as did achievement. Based on 
the observations we classified three classrooms as 
high-implementation classrooms, three as average- 
implementation, and three as low-implemention. At the 
conclusion of the field study, there was a 28- 
percentage-point difference between the high- 
implementing-classrooms mean gain score and that of the 
low-implementing classrooms. When covariance-ad justed 
postest means were considered, the difference was about 
20 percentage points. 



* This study was completed as a part of a grant funded 
by the United States Department of Education, project 
number 84.180R2 — Technology, Educational Media & 
Materials - Assessing the Social and Cultural Impact of 
Group-Based, Videodisc Math Instruction in Mainstreamed 
Classrooms. Its parent study was primarily concerned 
with teacher-student interactions and achievement 
outcomes for special education students in mainstreamed 
classrooms. The present study added the investigation 
of relationships between program implementation and 
achievement among regular education students. The 
studies were conducted concurrently, in the same 
classrooms. For a more complete description of the 
research conditions, see Lowry (1989). 



BACKGROUND 



Many ref.earchers of instructor-controlled 
instructional technologies try to ensure that the 
program is implemented as close to design 
specifications as possible, consistent with Mark's 
(1983) suggestions (e.g., Kelly, 1986). By doing so, 
the researchers can verify the treatment variable and 
so can argue that differences in the dependent variable 
are attributable to the program (see Shaver, 1983) . 

Media have been used in an attempt to 
"standardize" the delivery of instruction (Kemp and 
Smellie, 1989, p. 3) and so to reduce variations in 
implementation. But apparently, even when a well- 
defined, media-based program is put into the classroom 
and monitored, variations in implementation are 
observed. Hasselbring, Sherwood, and Bransford (1986) 
reported that they informally observed such variations 
during their study of the Mastering, Fractions vodeodisc 
program (Systems Impact, 1986) . 

In their study Hasselbring et al. (1986) were able 
to control the implementation of the program to some 
degree. Larger variations in program implementation 
are apt to occur when instructors use a program in 
their day-to-day work, outside the field-test 
environment (Shaver, 1964? Heinich, 1984). A number of 
authors have suggested that differences between what an 
instructional designer recommends and what is 
implemented in the classroom may have a significant 
effect on student achievement (e.g.. Stake, 1967; Hall 
and Loucks, 197 7) , 

It is important that program implementation, for 
pre-designed instructional programs, be gathered 
systematically (Shaver, 1983). Findings of such 
research can be used to evaluate the specific program. 
But information regarding implementation versus the 
learning outcomes of a program can also be used to 
evaluate instructional design/development in general. 
If an instructional design, based on theoretical and 
empirical foundations, is manifested in an 
instructional product; the product is utilized "well" 
in the classroom; and the results are positive; then 
one of the assumptions regarding instructional design 
is supported . That assumption is that "... 
systematically designed instruction can greatly affect 
individual human development" (Gagne, Briggs, & Wager, 
1988, p. 5; see also Dick & Carey, 1990). 
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Mastering Fractions was designed to be consistent 
with the Direct Instruction Model (Engeljnann and 
Carnine, 1982) and with the findings of the "effective 
teaching" literature (see e.g., Engelinann and Carnine, 
1982? Rosenshine and Stevens, 1986? Hofmeister & Lubke, 
1990) . Further, some of the features of the effective 
teacher are "captured" in the videodisc medium 
(Hofmeister, personal communication) . Therefore, the 
program was considered an example of a well-grounded 
and well-defined intervention, based on a systems 
approach to instructional design and development. 



PURPOSE OF THE RESEARCH 

To extend the work of Hasselbring et al. (1986), 
we measured program implementation level as the 
Mastering Fractions program was being used at three 
sites. The question specifically addressed by this 
study was as follows: Does the level of student 
achievement differ across levels of implementation of a 
mediated program that is based on a well-defined 
instructional model? 



METHOD 

First, to check the overall effect of the 
Mastering Fractions program, we used a quasi- 
experimental, noneguivalent control group design to 
compare the postest achievement of two groups of upper 
elementary students (N = 3 37? 171 treatment, 166 
control) . The treatment group received instruction in 
fractions via the teacher-directed, videodisc-based, 
Mastering Fractions program. The control group 
students received their normal grade-four or grade-five 
mathematics program, which did not include common 
fractions. This part of the study provided data 
regarding treatment vs. no-treatment conditions. The 
mean, treatment-group, covariance-adjusted, posttest 
score on a criterion-referenced achievement test of 
fractions skills and concepts was 6.0 standard 
deviations above that of the control group (the effect 
size =5.9 for raw gain scores). These effect sizes 
were determined using the student as the unit of 
analysis. We concluded that the program did provide 
solid instruction in specific fractions skills and 
concepts. 

In addition to the comparison between the 
treatment and control groups, we collected data on 
differences in achievement within the treatment group 
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(n - 171) by level of program implementation. This 
addressed the research question posed in the present 
study. 

Subjects 

The facilitators involved in this study were nine 
grade-five teachers at three sites. Each had a minimum 
of five years of teaching experience. 

The students were grade-five, public school 
students who had not received instruction in fractions. 
Their grade equivalent scores (based on the Iowa Test 
of Basic Skills or the Comprehensive Test of Basic 
Skills ) are comparable to those presented in the 
technical manuals for the instruments (see Lowry, 
1989) . 

Instrumentation 

The two variables measured for this study were 
criterion-referenced student achievement and teacher 
implementation of the program. The former was measured 
using a paper-and~pencil test? the latter was 
considered using a combination of measures. 



Ach i evement me a su re 

We used a criterion-referenced test, developed for 
previous research with Mastering Fractions , as a 
measure of fractions achievement. The test contains 57 
items covering fractions skills and concepts. Since it 
is criterion referenced, there were low scores and low 
variability among scores at pretest. The test-retest 
correlation for control-group students is +0.67. 



Level of implementation of 

the Mastering Fractions Pioaram 

Shaver (1983) suggests the use of multiple 
measures of implementation, including systematic 
observation. He defines systematic observation as "... 
the unitization of behavior and the classification of 
the units into previously defined categories,..." (p. 
4) . We used three measures of implementation: a 
systematic-observation instrument, a self-report 
teacher questionnaire regarding program modification, 
and classroom sei.ting charts. 
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Observational instrument . Although implementation 
observation instruments are available, Shaver (1983) 
points out that these instruments "... may not reflect 
validly the critical dimensions of the teaching methods 
to be investigated" (p. 4) . The developers of 
Mastering ^ Fractions provide specific suggestions for 
implementing the program. These suggestions are 
presented in the teacher's manual. We combined the 
suggestions from the manual with two other sources; (1) 
information from informal observations made by 
experienced observers from earlier studies of the 
program and (2) analyses of videotapes of teachers who 
had used the program successfully in the past. Once 
the behaviors were listed, we categorized them 
according to a media utilization model (Kemp & Smellie, 
1989) and the effective teaching literature (see 
Hofmeister & Lubke, 1990) , The result was a dual- 
purpose listing. We used the list as a rating 
instrument for systematic observation and as a set of 
suggestions for teachers implementing the program. The 
suggestions sheet is presented in Appendix A; the 
rating sheet appears in Appendix B. 

During the course of the Mastering Fractions 
program in each of the 9 treatment-group classrooms in 
the fall of 1988, observers for the parent project were 
asked to record their perceptions of the degree of 
implementation of each area of the utilization model 
(see Appendix B) . The sum of the rating points (on a 
five-point scale; 5 as high), for all of the categories 
in the utilization model, was taken as an 
implementation score on this instrument. 

At least two observers would visit a single class 
together, once per week, to check interobserver 
agreement. One investigator also visited each 
classroom where the videodisc program was being used, 
to check interobserver agreement. Percentage of exact 
interobserver agreement was 75.6; agreement within one 
step on the rating scale was 96.7%, The latter 
percentage is provided, since it is quite conceivable 
that, on a five-point scale representing a continuous 
variable, one observer might perceive a behavior as 
just less than say 2.5 out of 5 (and so would record a 
score of 2) while another observer might see the same 
behavior as slightly more than 2.5 (and so would record 
it as a score of 3) . 

Teacher questionnaire . After each of the classes 
had completed the Mastering Fractions program, site 
coordinatorf> interviewed the respective teachers with a 
structured questionnaire as a self report of 
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modifications to the program. A copy of the 
questionnaire appears in Appendix C. The questionnaire 
provided a second indication of how teachers modified 
the recommended delivery practices. 

Seating charts . The teacher's guide also includes 
suggestions for seating patterns, to allow the teacher 
to provide needed help to students who may be 
experiencing difficulties with the program. The 
teachers provided seating charts to the observers, so 
that the observers could identify students, by name, 
for the parent-study observations. We used the seating 
charts to determine the degree to which teachers were 
able to arrange the class seating in accord with the 
suggestions. 

We then compiled the data to get an overall- 
implementation consistency rating for each teacher, 
based on the questionnaire, the implementation 
observations, and the seating charts. 



The Instructional Programs 



Teacher in-service instruction 

Treatment-group teachers were given a half -day 
workshop on the use of the program, and were invited to 
call the researchers with their questions or comments. 
This procedure is consistent with the typical in- 
service training for the program. 

During each of the teacher-training sessions we 
provided copies of the suggestions sheet (Appendix A) 
and discussed each item on the list. We demonstrated 
the set-up and use of the equipment. l<7e addressed each 
of the program utilization techniques by showing and 
discussing a videotape of a model teacher using the 
program. We encouraged teachers to follow the 
suggestions as closely as possible. 



The Mastering Frac tions program 

Rosenshine and Stevens [1986) provide a list of 
teaching processes in their operational definition of 
the daily functions of direct instruction; 

1. Review, check the previous day's work (and 
reteach, if necessary) 

2. Present new content/ skills 
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3. Guide student practice (and check for 
understanding) 

4. Provide feedback and correctives (and reteach, 
if necessary) 

5. Provide independent student practice 

6. Provide weekly and monthly reviews. 

But rote following of these instructional functions 
does not guarantee that students will learn. 
Rosenshine and Stevens (1986) noted that the success of 
these techniques depends on specific ways of performing 
each of them. For example, content or skills should be 
presented in a clear and organized manner (item 2 
above) . Haphazard presentation does not benefit the 
learner (see also Bandura, 1982) . 

Direct forms of instruction have been observed by 
researchers studying effective-teaching in practice 
(e.g, Good, 1979). But the Direct Instruction Model 
(Engelmann, 1980) has evolved from a 

theoretical/empirical base. For a detailed description 
of the model see Engelmann and Gamine (1982) (see also 
Becker & Carnine, 1980). 

The Mastering Fractions program was developed in 
cooperation with Engelmann and Carnine. it was 
designed to be presented to a heterogeneous class of 
students. The program requires that a television or 
monitor, with a screen large enough to be read in all 
parts of tne classroom, be placed at the front of the 
class. All students are seateu facing the screen. The 
audio system must be clear and loud enough to be heard 
by all students. The program requires the Mastering 
Fractions videodiscs, a level-one videodisc player, 
consumable student worksheets, extra paper and pencils, 
and the teacher handbook (which includes answer keys 
for the seat work and tests) . 

Mastering Fractions consists of 35 lessons. Each 
lesson is designed to be completed in approximately 40 
minutes. Most presentation/practice lessons begin with 
a quiz of the previous lesson's learning, and each 
fifth lasson is a mastery test. Both quizzes and 
mastery tests are followed by diagnostic/remedial 
suggestions based on specific skill weaknesses 
manifested by the students. Instruction is provided on 
the disc at three progressive levels — oral response, 
written practice with component skills and concepts, 
and written practice with articulated skills and 
problem-solving strategies (for more detail see Systems 
Impact, 1986) . 
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The developers of the Direct Instruction Model 
have provided specific guidelines for the presentation 
and practice of concept attainment and skill 
performance. That is, general definitions of direct 
forms of instruction address instructional management 
or a superstructure for instruction, while the Direct 
Instruction Model also provides instructional design 
rules and message design guidelines. The videodisc- 
based portion of the Mastering Fractions program was 
designed using the D. I. Model, and the paper-based 
materials and recommended teaching procedures are 
designed to support that model. However, most 
important to the present study, while the program is 
media based, it is teacher controlled. The designers 
purposely gave the teacher control over portions of the 
program to help integrate Mastering Fractions into the 
school context. 



Analysis 

Each teacher did modify the delivery of the 
Mastering Fractions program to some degree. When the 
treatment-group classes had finished the program, the 
observations were averaged for each teacher on a per- 
observation-period basis. The post-study questionnaire 
data and seating chart data were also included. The 
seating charts showed that none of the teachers used 
the suggested seating arrangement consistently. Scores 
on the post-study questionnaire were consistent with 
the observational data. We then classified the classes 
as high-implementing, average-implementing, and low- 
implementing — three classes in each subgroup. 

Adequate analysis was made by simple inspection of 
the data (Shaver, 1983) (see RESULTS AND DISCUSSION) . 
We also ran an analysis of covariance, using pretest 
means on a standardized achievement measure and pretest 
means on the criterion-referenced test as covariates. 



RESULTS AND DISCUSSION 
^Achievement ar^c^ prpgr^^l^ XffP^eTnent^^tion 

Table 1 shows mean percentage scores and gain 
scores on the fractions criterion-referenced test. The 
treatment-group means are broken down by the level of 
program implemenation. The the no-treatment control 
group means are also included for your information. 
The overall m^ean of the treatment group is also 
included. 



ERIC 



c • 

4oy jQ 



10 



Table 1 



Percentage Correct Means on the Criterion-Referenced 
Achievement Test bv Program Implementation Level for 
the Treatment Group 



Implementation 

Level Pretest Posttest Gain n-size 



SD X SD 



High 6.6 5.0 84.9 10.0 +78.3 55 

Average 11.5 15.3 79.7 14.6 +68.2 54 

Low 8.6 11.2 58.9 24.8 +50.3 62 

Control 10.6 10.7 9.2 10.6 -01.4 166 



overall 
within 

the 8.9 11.4 73.8 21.3 +64.9 171 

treatment 

group 



As would be expected the control group scores show 
essentially no change on the fractions criterion- 
referenced test from from pre- to posttest. The 
treatment group means, however, show large gains (64.9 
percentage points on the average) . As noted earlier, 
overall, the results of the criterion-referenced test 
provide evidence that students who study the y^asteyina 
Fractions program do learn specific fractions skills 
and concepts. The standardized mean difference effect 
size between the treatment and control groups was +5,9 
in favor of the treatment group. The mean scores on 
the criterion test were also consistent with gains made 
in previous research on this instructional system 
(e.g., Hasselbring et al,, 1986), 

There are also clear differences among the means 
for high, average, and low levels of implementation. 
There is a 10 percentage-point difference between the 
high- and average-implementation mean gain scores, and 
an 18 percentage-point difference between the average- 
and the low-implementation mean gain scores. 




The analysis of covariance yielded an effect of 
levels-of-implementation that is statistically 
significant when the student is used as the unit of 
analysis (F = 25.92; e ~ .001). The eta-squared effect 
size, based on the adjusted means, is +0.12, which 
means that 12% of the variance in the criterion- 
referenced posttest scores is associated with 
differences in implementation level. Table 2 shows the 
covariance~ad justed posttest means for the criterion- 
referenced posttest scores (for details see Lowry, 
1989) . There remained a 20-percentage point spread 
between the mean of the high-implementation classes and 
that of the low-implementation classes. No site 
dominated any of the implementation levels. 

Table 2 

Summary of Adjusted Cell Means for the AnalvsiR 
of Covarj.ance 



Level of Mean 
Implementation 



High 


83. 


6% 


Average 


76. 


1% 


Low 


63. 


9% 



Discussion 

The findings of this study are consistent with 
observations made by Hasselbring et al. (1986). 
Variations in implementation/utilization did occur with 
an instructional system that provides for some autonomy 
on the part of the teacher. 

Further, the achievement on a criterion-referenced 
test appears to be directly related to the level of 
implementation of the program. This relationship 
supports an assumption that systematically designed 
instruction can affect individual human development. In 
this case the area was achievement enhancement, the 
goal of a direct-instruction-type program. 

The results also support the position that 
systematic design is necessary but insufficient. If a 
causal relationship exists between levels of 
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implementation of a weal 1 - designed program and 
achievement, for this or other systems-designed 
programs, designers must be cautious. If an 
instructional system is too sensitive to the contaxt in 
which it is implemented, i.e., too open, its potential 
effect may be waterod-down (as noted by Heinich, 1984). 
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Appendix A 



SUGGESTIONS FOR THE USE OF 
•KiASTER(NG FRACTIONS^ 



9/20/88 



Preparing Yourself 

- praclice setting up Ihe equipment. 

• practice using the program Detore you teach with it. 

- took for subtle variations. 

- go thrnngh the wriia^check screens and the worksheets. 

- practice with the hand controller. 

- took for potential problem areas fcr your students. 

Preparing Your Environment 
" ensure the equipment is ready. 

• ensure the workbook pages are photocopied. 

• ensure the lighting level is appropriate. 

• ensure there is no giare on the TV screen. 

- ensure all students can read the screen and hear the audio track. 

• ensure there is adequate space for you to walk around the room. 

Preparing the Students 

- note the lesson number for today aloud. 

- cai! their attention to the information on the screen (help them focus on the screen) 

- ensuie students have paper, sharp pencils, etc. 

- position students having difficulty near the cenier of the group. 

• stop disnjptions. 

- introduce suppon skills. 

• support the program vertally and by your actions. 

Evaluating Previous Learning 

- use the review quiz. 

• allow adequate work time for each problem. 

- use quiz results to determine remediation. 

- use the *1/5 cnterion* 

• use progress sheets. 

- aitow student mastery, not a weekly schedule, to determine program pacing. 

Presenting and Remediating the tnstructtonai Material 

• circutate among the students. 

• model/encourage appropriate student verba! response. 

- use the scan and step functtons to ireete frame or review/reiterate concepts or vocabulary, 

• stop the program to clarify or enhance the instructton if students are having difficulty. 

Guiding Practice 

• drcuiate to check student progress on their work products. 

- diagnose student problems. 

- coach students, a step at a time, through new types of problems, using previously teanned skills. 

- keep the pace moving. 

• have alternate work available for high achieving students (they can switch their attention 
from math to another project and back, as necessary). 

- wofK briefly with individuals, then circulate: review/remediate with the group as 
necessary. 
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ls:plct3enLac iop Rating Sheet 



SUGGESTIONS FOR THE USE OF 
•NV^TERING FRACTIONS" 



9/20/88 



Preparing Younseff 



- pfaciice setting up the equipment. 

- practice using the program before you teach with ii. 

- took for subtle variations. 

• go through the write/check screens and the worksheets. 

- practice with the hand controller. 

- look for potential problem areas for your students. 

Preparing Your Environment 1 2 3 ,4_5 

• ensure the equipment is ready. 

• ensure the workbook pages are photoccpied. 

- ensure the lighting level is appropriate. 

- ensure there is no glare on the TV screen. 

" ensure all students can read the screen and hear the audio track. 

• ensure there is adequate space for you to walk anound the room. 

Preparing the Students 1 2 3 4 5_ 

- note the lesson number for today aloud. 

- call their attention to the information on the screen (help them focus on the screen) 

- ensure students have paper, sharp pencils, etc. 

- position students having difficulty near the center of the group. 

- stop disruptions. 

- introduce support skills. 

- support the program verbally and by your actions. 

Evaluating Previous Learning 1 2 3 4 5 

- use the review quiz 

• allow adequate work time for each problem. 

- use quii results to determine remediation, 
-use the'l/Scfiterion" 

- use progress sheets. 

- allow student mastery, not a weekly schedule, to determine program padng. 

Presenting and Remediating the Instructional Materia! 1 2 3 4 5 

- circulate among the students. 

- model^encourage appropriate student verba! response. 

• use the scan and step functions to freeze frame or review/reiterate concepts or vocabulary. 

• stop the program to darify or enhance the instruction if students are having difficulty. 

Guiding Practice 1 2 3 4 5 

• circulate to check student progress on their work products. 

• diagnose student problems. 

- coach students, a step at a time, through new types of problems, using previously learned skitts. 

- keep the pace moving. 

- have alternate work available for high achieving students (they can switch their attention 
from math to another project and back, as necessary). 

- work briefly with indrviduais, then circulate; review/remediate with the group as 
necessary. 
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Teacher Questionnaire 

Teacher 



Date of Interview 



During y ,r post- ij^s terjn3_^ Fr act i on s interview^ we collected 
information regardTng^Jour TieTTngs about the program. The 
program developers are also interested in how you used Mjsterinc 
fractions. They would like any information on how you maJ'Tiive^ 
mod"T?Tid"the program. 

I. How many weeks do you nor.Tially spend on fractions, at' the 
grade level you're teaching this year? 



week s 



2. Was there terminology or content in the Mastering Fractions 
program which you modified. If so, what* s peel?! c"'terms~or~ 
concepts stand out in your mind? 



3. Were there instructional nethods used in the program which 
you modified? Specifically 



Did you modify the 1/5 criterion for remediation, which 
appears on the "CSP" screens? If so, how? 



b. Did you require students to "write" or "copy" problems 
where the program asked students to 'copy the problem and 
work it"? If not, what other 8pproach{es) did you use? 
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Did you require choral responses during the verbal 
practice presented in the program? 



d* Did you use choral responses in place of written 
responses? In what parts of th^ program? 



Did the students use the workbooks as homework or did they 
use them as 1 "dependent $ gatwor k ? ( PI ia se^'cTrc 1 e one or both 
- if both appTyT^t nine's :a*Ee the approximate percent of time 
for each.] 



5. Did you supplement the videodisc material with other 

materials? If so» dirring what parts of the program^ or 
regard Ing wi^at concepts / strategies? 



6* During the Fall session, USU Staff developed "Suppl ements'' 
intended to help students learn multiple labels for the 
concepts they had learned uuring Ha s ter i no Fractions ; that 
is, to generalize the concepts they'^wer i^t earni ngT 

Did you use the Supplements? ^ ^ 

If so, do you think your students benefited frotn their use? 



47 B 
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7. Oo you think that H a s t e r 1 n o Fraction s Is appropriete for the 
grade level you teacn? " 

If not, what recommendations would you have for its use? 



8. Was there a point In the Mas ter i ng Fr a c 1 1 on s where yocr 
lowest students Degan to experience c i tTTcuI ty? 

If so, at about what lesson? 
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